…nalytical-validation-framework-and-elasticity-sensitivity-analysis

Implements utils/post/diagnose_demand_flex.py: an analytical diagnostic
that loads real HP building loads and TOU derivation data to sweep
candidate elasticity values (-0.02 to -0.20), compute peak reduction %
and rate-arbitrage savings per HP building, and recommend the elasticity
that best matches the Arcturus 2.0 "no enabling technology" empirical
target for each NY utility.

Runs in ~15s per utility without requiring a CAIRO run.

Made-with: Cursor

Implements utils/post/validate_demand_flex_shift.py: reproduces the
constant-elasticity demand-response shift analytically using the same
functions CAIRO uses, then verifies correctness via three checks:
energy conservation (per-building kWh preserved), direction (peak kWh
decreases, off-peak increases), and cross-check against CAIRO's saved
demand_flex_elasticity_tracker.csv. All 7 NY utilities pass at ε=-0.10.

Produces five diagnostic plots per utility: aggregate daily load
profile (pre/post with shaded fill), net shift by hour bars, month×hour
heatmap, per-building peak reduction distribution, and per-building
daily profiles at representative consumption percentiles. Also writes
CSV/parquet data outputs for report reuse.

Made-with: Cursor

context/methods/tou_and_rates/demand_flex_elasticity_calibration.md:
accessible writeup of the per-utility elasticity calibration methodology,
Arcturus 2.0 empirical anchor, two savings mechanisms (rate arbitrage vs
RR reduction), results table (ε=-0.10 or -0.12 per utility), and known
limitations.

context/plans/demand-flex_elasticity_calibration.md: living plan document
for the broader demand-flex calibration and validation work.

Made-with: Cursor

rate_design/hp_rates/ny/Justfile: add validate-demand-flex,
validate-demand-flex-all, and diagnose-demand-flex recipes for running
the shift validation and elasticity sweep diagnostic from the command
line.

context/README.md: index the new demand_flex_elasticity_calibration.md
context document.

Made-with: Cursor

utils/cairo.py and utils/demand_flex.py now accept a season-keyed dict
for the elasticity parameter in addition to a scalar. The shifting
pipeline resolves the per-season value inside _shift_season so summer
and winter can have independently calibrated epsilons without any
structural change to the call sites.

Made-with: Cursor

run_scenario.py now passes the elasticity value from the scenario YAML
directly to the CAIRO runtime unchanged (dict or scalar), enabling
seasonal elasticity to flow from config/periods/*.yaml through
create_scenario_yamls into the actual CAIRO run.

Made-with: Cursor

Moves the elasticity calibration script from utils/post to utils/pre
since its primary output is config/periods/*.yaml, which is consumed
before a CAIRO run. Changes from the original:

- Dual Arcturus model: computes both no-tech (slope=-0.065) and
  with-tech (slope=-0.111, corrected interaction term) recommendations
- Seasonal elasticities: matches each season's price ratio against its
  own Arcturus target independently
- --write-periods: writes both elasticity and elasticity_with_tech dicts
  to config/periods/{utility}.yaml
- --compare-batch / --with-tech: compares analytical savings against a
  completed CAIRO batch, selecting the right reference set
- --epsilon-start/--epsilon-end/--epsilon-step: replaces hardcoded list
  with a configurable sweep range (default -0.04 to -0.50, step -0.02)
- ty/ruff fixes: cast for period_rate, .to_series().dt.month.to_numpy()
  for DatetimeIndex.month, removed dead assignments

Made-with: Cursor

…ing_tech

_row_to_run now reads enabling_tech from the Google Sheet row and selects
elasticity_with_tech from the periods YAML when enabling_tech is empty,
true, yes, or 1 (with-tech is the default). Only explicit false/no/0
selects the no-tech elasticity key. This makes enabling technology opt-out
rather than opt-in, matching the intended workflow.

Made-with: Cursor

json.dumps() does not append a newline, causing every _calibrated.json
to fail the end-of-file-fixer pre-commit hook and show spurious diffs.

Made-with: Cursor

- Add comprehensive usage examples (just recipes and direct uv run) to
  the module docstring covering scalar, seasonal, no-tech, with-tech,
  and CAIRO cross-check invocations
- Fix DatetimeIndex.month access: use .to_series().dt.month.to_numpy()
  to avoid ty unresolved-attribute and pandas alignment errors
- Cast period_rate to pd.Series to satisfy ty invalid-assignment check
- Remove unused bldg_level assignment and dead p_flat groupby block
- Remove stale type: ignore[arg-type] comments
- Fix set_xticklabels: pass list[str] instead of range

Made-with: Cursor

diagnose_demand_flex.py, sensitivity_demand_flex.py, and
validate_demand_flex.py are replaced by calibrate_demand_flex_elasticity
(now in utils/pre) and validate_demand_flex_shift. The corresponding
test file is also removed as it tested the deleted validate_demand_flex.

Made-with: Cursor

- Move module reference from utils.post to utils.pre
- Add comprehensive usage examples for all invocation patterns
  (default sweep, utility subset, custom range, --write-periods,
  --compare-batch with and without --with-tech)

Made-with: Cursor

Full calibration sweep (epsilon -0.04 to -0.50, step -0.02) across all
7 NY utilities. Each periods YAML now has:
  elasticity: {summer: ..., winter: ...}       # Arcturus no-tech match
  elasticity_with_tech: {summer: ..., winter: ...}  # Arcturus with-tech match

No-tech summer epsilons range from -0.10 (NiMo/NYSEG/RGE) to -0.14
(CenHud/OR/PSEG-LI). With-tech values are ~1.5-1.7x larger. Seasonal
elasticities independently match each season's price ratio against the
Arcturus target rather than using a single annual approximation.

Made-with: Cursor

Scenario YAMLs regenerated from the Google Sheet via create_scenario_yamls.
Demand-flex runs now carry a seasonal elasticity dict read from the utility's
periods YAML (elasticity_with_tech when enabling_tech is empty or true,
elasticity otherwise). This replaces the previous scalar elasticity value
and enables summer/winter-specific load shifting in CAIRO.

Made-with: Cursor

… run

Promoted calibrated tariff JSONs from the latest CenHud CAIRO run. All
files now end with a trailing newline (fixed in copy_calibrated_tariff_from_run).

Made-with: Cursor

… full results

- demand_flex_elasticity_calibration.md: corrected Arcturus with-tech
  coefficients (interaction term: slope=-0.065 + -0.046 = -0.111),
  updated results tables with full 7-utility seasonal sweep for both
  no-tech and with-tech, refreshed invocation examples, added note on
  enabling_tech defaulting to with-tech
- context/README.md: index updated to reflect calibration script move
  to utils/pre

Made-with: Cursor

Made-with: Cursor